{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model monitoring and drift detection\n",
    "\n",
    "This tutorial illustrates leveraging the model monitoring capabilities of MLRun to deploy a model to a live endpoint and calculate data drift.\n",
    "\n",
    "Make sure you have reviewed the basics in MLRun [**Quick Start Tutorial**](../01-mlrun-basics.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tutorial steps:\n",
    "- [**Create an MLRun project**](#setup-project)\n",
    "- [**Log a model with a given framework and training set**](#log-model)\n",
    "- [**Import and deploy serving function**](#import-deploy)\n",
    "- [**Simulate production traffic**](#simulate-traffic)\n",
    "- [**View drift calculations and status**](#view-drift)\n",
    "- [**View detailed drift dashboards**](#view-dashboards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLRun installation and configuration\n",
    "\n",
    "Before running this notebook make sure `mlrun` is installed and that you have configured the access to the MLRun service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Install MLRun if not installed, run this only once (restart the notebook after the install !!!)\n",
    "%pip install mlrun tqdm ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup-project\"></a>\n",
    "## Set up the project\n",
    "\n",
    "First, import the dependencies and create an [MLRun project](https://docs.mlrun.org/en/latest/projects/project.html). This  contains all of the models, functions, datasets, etc.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import mlrun\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-03-12 17:02:37,120 [info] loaded project tutorial from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "project = mlrun.get_or_create_project(name=\"tutorial\", context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} Note\n",
    "This tutorial does not focus on training a model. Instead, it starts with a trained model and its corresponding training dataset.\n",
    "```\n",
    "\n",
    "Next, log the following model file and dataset to deploy and calculate data drift. The model is a [AdaBoostClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html) from sklearn, and the dataset is in `csv` format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We choose the correct model to avoid pickle warnings\n",
    "import sys\n",
    "\n",
    "suffix = (\n",
    "    mlrun.__version__.split(\"-\")[0].replace(\".\", \"_\")\n",
    "    if sys.version_info[1] > 7\n",
    "    else \"3.7\"\n",
    ")\n",
    "\n",
    "model_path = mlrun.get_sample_path(f\"models/model-monitoring/model-{suffix}.pkl\")\n",
    "training_set_path = mlrun.get_sample_path(\"data/model-monitoring/iris_dataset.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"log-model\"></a>\n",
    "## Log the model with training data\n",
    "\n",
    "Log the model using MLRun experiment tracking. This is usually done in a training pipeline, but you can also bring in your pre-trained models from other sources. See [Working with data and model artifacts](https://docs.mlrun.org/en/latest/training/working-with-data-and-model-artifacts.html) and [Automated experiment tracking](https://docs.mlrun.org/en/latest/concepts/auto-logging-mlops.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"RandomForestClassifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_artifact = project.log_model(\n",
    "    key=model_name,\n",
    "    model_file=model_path,\n",
    "    framework=\"sklearn\",\n",
    "    training_set=pd.read_csv(training_set_path),\n",
    "    label_column=\"label\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'store://models/tutorial-yonis/RandomForestClassifier#0:9e8859ee-dc11-4874-a4f7-ebdce46a5a82'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the model artifact unique URI\n",
    "model_artifact.uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"import-deploy\"></a>\n",
    "## Import and deploy the serving function\n",
    "\n",
    "Import the [model server](https://github.com/mlrun/functions/tree/master/v2_model_server) function from the [MLRun Function Hub](https://www.mlrun.org/hub/). Additionally, mount the filesytem, add the model that was logged via experiment tracking, and enable drift detection.\n",
    "\n",
    "The core line here is `serving_fn.set_tracking()` that creates the required infrastructure behind the scenes to perform drift detection. See the [Model monitoring overview](https://docs.mlrun.org/en/latest/monitoring/model-monitoring-deployment.html) for more info on what is deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Import the serving function from the Function Hub and mount filesystem\n",
    "serving_fn = mlrun.import_function(\"hub://v2_model_server\", new_name=\"serving\")\n",
    "\n",
    "# Add the model to the serving function's routing spec\n",
    "serving_fn.add_model(model_name, model_path=model_artifact.uri)\n",
    "\n",
    "# Enable model monitoring\n",
    "serving_fn.set_tracking()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the serving function with drift detection\n",
    "\n",
    "Deploy the serving function with drift detection enabled with a single line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-03-12 17:02:38,651 [info] Starting remote function deploy\n",
      "2023-03-12 17:02:40  (info) Deploying function\n",
      "2023-03-12 17:02:40  (info) Building\n",
      "2023-03-12 17:02:40  (info) Staging files and preparing base images\n",
      "2023-03-12 17:02:40  (info) Building processor image\n",
      "2023-03-12 17:03:40  (info) Build complete\n",
      "2023-03-12 17:03:51  (info) Function deploy complete\n",
      "> 2023-03-12 17:03:52,969 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-tutorial-yonis-serving.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['tutorial-yonis-serving-tutorial-yonis.default-tenant.app.vmdev30.lab.iguazeng.com/']}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://tutorial-yonis-serving-tutorial-yonis.default-tenant.app.vmdev30.lab.iguazeng.com/', 'name': 'tutorial-yonis-serving'})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlrun.deploy_function(serving_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View deployed resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you should see the newly deployed model server, as well as a `model-monitoring-stream`, and a scheduled job (in yellow). The `model-monitoring-stream` collects, processes, and saves the incoming requests to the model server. The scheduled job does the actual calculation (by default every hour).\n",
    "\n",
    "```{admonition} Note\n",
    "You will not see `model-monitoring-batch` jobs listed until they actually run (by default every hour).\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![drift_table_plot](../_static/images/tutorial/project_dashboard.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"simulate-traffic\"></a>\n",
    "## Simulate production traffic\n",
    "\n",
    "Next, use the following code to simulate incoming production data using elements from the training set. Because the data is coming from the same training set you logged, you should not expect any data drift.\n",
    "\n",
    "```{admonition} Note\n",
    "By default, the drift calculation starts via the scheduled hourly batch job after receiving 10,000 incoming requests.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ed9279ceb6f3489a95ad327bf23118c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/12000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json\n",
    "import logging\n",
    "from random import choice\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# Suppress print messages\n",
    "logging.getLogger(name=\"mlrun\").setLevel(logging.WARNING)\n",
    "\n",
    "# Get training set as list\n",
    "iris_data = (\n",
    "    pd.read_csv(training_set_path).drop(\"label\", axis=1).to_dict(orient=\"split\")[\"data\"]\n",
    ")\n",
    "\n",
    "# Simulate traffic using random elements from training set\n",
    "for i in tqdm(range(12_000)):\n",
    "    data_point = choice(iris_data)\n",
    "    serving_fn.invoke(\n",
    "        f\"v2/models/{model_name}/infer\", json.dumps({\"inputs\": [data_point]})\n",
    "    )\n",
    "\n",
    "# Resume normal logging\n",
    "logging.getLogger(name=\"mlrun\").setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"view-drift\"></a>\n",
    "## View drift calculations and status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once data drift has been calculated, you can view it in the MLRun UI. This includes a high-level overview of the model status:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model_endpoint_1](../_static/images/tutorial/model_endpoint_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more detailed view on model information and overall drift metrics:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model_endpoint_2](../_static/images/tutorial/model_endpoint_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as a view for feature-level distributions and drift metrics:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![model_endpoint_3](../_static/images/tutorial/model_endpoint_3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"view-dashboards\"></a>\n",
    "## View detailed drift dashboards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, there are also more detailed Grafana dashboards that show additional information on each model in the project:\n",
    "\n",
    "For more information on accessing these dashboards, see [Model monitoring using Grafana dashboards](https://docs.mlrun.org/en/latest/monitoring/model-monitoring-deployment.html#model-monitoring-using-grafana-dashboards)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grafana_dashboard_1](../_static/images/tutorial/grafana_dashboard_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphs of individual features over time:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grafana_dashboard_2](../_static/images/tutorial/grafana_dashboard_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as drift and operational metrics over time:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grafana_dashboard_3](../_static/images/tutorial/grafana_dashboard_3.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
